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ABSTRACT 

In this paper, we describe the successful implementation of an afterschool LEGO robotics 
program for elementary and middle school students that is annually offered by the Whitacre 
College of Engineering at Texas Tech University. Three events are held on campus: the kickoff, a 
trial run, and the competition, spread over a period of eight weeks. In between the events, 
participants design their LEGO robots at the school, mostly during afterschool clubs. Through 
our program, we broaden the participation in hands-on robotics tasks which apply STEM 
concepts to groups that otherwise might not have the opportunity. Success factors of our 
implementation are the flexibility of the implementation at the local level, the inclusion of 
engineering students as mentors and volunteers, and the low cost for organizers and participants. 

We provide evidence that we have reached a diverse student population in grades K-8 and 
positively changed their attitudes toward STEM, then we report the benefits that teachers see in 
regular participation in the event. Finally, we describe the benefit of involving engineering 
undergraduate students as mentors and volunteers. 

Keywords: LEGO Robotics Competition, K-8; Reaching Underserved Student Population; Afterschool Programs 


1 INTRODUCTION 


he field of robotics is interdisciplinary per se and thus lends itself to teach science, technology, 
engineering, and math (STEM) concepts in an interconnected way and through hands-on experiential 
learning. In addition, many robotics kits, such as the LEGO Mindstorms NXT kits, remind children 
of toys they know from at home and thus immediately engage them in a playful way and keep them engaged for 
extended time periods (Mauch, 2001) 



Robotics programs have been proven to be successful tools to engage students in K-12 in STEM and create 
interest in careers in the STEM field (Hobson, 2000; Mitchell, Warwick, Browne, Gasson, & Wyatt, 2010; Habash, 
Suurtamm, 2010) that is then supported by a variety of scholarships available to college applicants with robotics 
experience. 1 Robotics has also been recognized as a topic that easily excites college level engineering students, 
offers hands-on experience to support theoretical concepts, and fosters multidisciplinary work (Mitchell et ah, 2010; 
Grimes & Seng, 2008; Caron, 2010). LEGO robotics in particularly has seen a wide implementation in K-12 as an 
exciting, hands-on way to apply and reinforce science concepts (Franz & Elmore, 2009; Williams, Ma, Prejean, 
Ford, & Lai, 2007) and improve problem solving skills (Mauch, 2001; Robinson, 2005). It has also been 
successfully used in preservice teacher education (Chambers and Carbonaro 2003). Nugent, Barker, & Grandgenett 
(2012) showed the effectiveness of using LEGO robotics to teach science, engineering, and technology concepts in 
after school 4-H clubs in rural Nebraska with participants aged 9-11 years old. 


1 FIRST Robotics Scholarships, https;//mv.usfirst.org/scholarshit>s/index.lasso?page=scholarshipsearch printed , accessed 
February 11, 2013. 
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This paper describes an afterschool robotics challenge. Get Excited About Robotics (GEAR), which is 
annually offered by the Whitacre College of Engineering (WCOE) at Texas Tech University (TTU) with the goal of 
exciting young elementary and middle school students about careers in STEM and engineering in particular through 
hands-on challenges that foster problem solving skills, team work, critical thinking, and self-efficacy. Over the last 
years this program has seen significant growth in participation, particularly among schools from small school 
districts in rural areas. As opposed to other robotics challenges, the program is not stand-alone but connects 
teachers and participants with engineering undergraduate students and faculty. 

We will report evidence of a successful implementation that reaches out to a diverse participant body and a 
large percentage of Title 1 schools, i.e. schools receiving funds under the Title I of the Elementary and Secondary 
Education Act (1965) to improve academic achievement of the disadvantaged. In other research. Title 1 status has 
generally been accepted as an indicator of low socioeconomic status for students (Bardack, Seidel, Seiter, & 
Lampron, 2011). We also report benefits of participation as stated by teachers, and the format in which engineering 
students are involved. 

While the best case scenario would be to have evidence that participation in GEAR increases the 
participants’ performance in STEM disciplines and their likelihood to choose a career in this field, this is beyond the 
scope of this paper due to chronological constraints. Additionally, selection criteria at the participating schools are 
too different to ensure that students who have a continued interest in STEM also have the opportunity of continued 
participation in GEAR and other robotics programs. Also, student self-selection is a possible confounding factor: at 
most schools, students must choose to take part in this program, and it is likely that the populations of students who 
do so materially differ in important ways from those who do not, e.g. they might already have a higher interest in 
STEM activities. Therefore, it is most appropriate to observe effects of this program on students while they 
participate in order to observe immediate factors in a low-cost and authentic manner. 

2 GET EXCITED ABOUT ROBOTICS (GEAR) 

GEAR 2 is a 6-8 week LEGO robotics challenge for students in elementary and middle school currently 
offered at several locations in Texas 34 on an annual basis. It uses the LEGO MINDSTORMS NXT kits 5 , which 
include a programmable brain, motors, several sensors (sound, light, touch, ultrasonic) in addition to typical LEGO 
bricks, beams, connectors, and axles. The theme of the competition changes every year and is motivated by real 
world applications of robotics. It is developed and provided by a 501 (c) 3 volunteer organization 6 . Previous years' 
themes included a science station in Antarctica, automated farming, and the maintenance of energy infrastructure. 
On game day at the end of the 6-8 weeks period, teams compete on an 8’ x 8’ table in two minute rounds during 
which they need to accomplish as many of the tasks as possible, each of which is associated with a score related to 
its difficulty. While GEAR is in many aspects similar to Junior FIRST LEGO League 7 , the major difference is that 
it empowers the tournament organizers by providing flexibility in the organization to best meet local needs. The 
general GEAR rules are 12 pages, which include all judging and scoring rules, and a description of all types of 
awards. The annual challenge description is of similar length and includes a description of all tasks, the scores 
associated with them, and instructions on where to buy game pieces and how to assemble them. Degrees of 
flexibility for the organizers include timing of the competition, number of teams allowed per school, fees charged, 
and services provided by the organizers to the participants. 

Our GEAR tournament has been hosted by the Whitacre College of Engineering since 2006 and has seen 
continued growth in participation and continued financial support from various entities. The competition typically 
kicks off during Engineers’ Week 8 in February and runs through the middle of April with the competition being held 


“ Get Excited About Robotics (GEAR), http://www.gearrobotics.org . accessed February 2013. 

3 Get Excited About Robotics (GEAR) at Texas Tech University, http://www.coe.ttu.edu/steni/gear/ , accessed February 2013. 

4 Get Excited About Robotics, University of Texas in San Antonio, http://itec.utsa.edu/ , accessed February 2013. 

5 LEGO MINDSTORMS, http://www.mindstorms.lego.com/ , accessed February 2013. 

6 Get Excited About Robotics (GEAR), http://www.gearrobotics.org . accessed February 2013. 

7 FIRST LEGO League (FLL), http://firstlegoleague.org/ . accessed February 2013. 

3 National Engineers Week Foundation, http://www.eweek.org/ . accessed February 2013. 
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during National Robotics Week. 9 It consists of three events all held on the TTU campus: During the kickoff event, 
the annual challenge is revealed and related to real-world engineering problems through short presentations given by 
professionals. Each school receives a set of game pieces, a game mat, and the rules. Participants then start 
designing, programming, and testing their robots at school, mostly in afterschool programs. However, some schools 
also incorporate the competition into elective courses such as technology or robotics, or offer it as part of the Texas 
Gifted and Talented (GT) program. Participants meet 1-5 times per week for 30 - 90 minutes each. Participants 
return to campus for a trial run under real competition settings toward the end of March. This event provides them 
with the opportunity to learn from other teams and evaluate their performance. Finally, teams return to campus after 
eight weeks for game day when they compete for awards. The competition consists of four seeding rounds and then 
finals in which the eight highest ranked teams participate. In addition to first to fourth place awards, there are a 
variety of other awards ranging from best themed team to a young engineering award 10 . 

Participation in the TTU GEAR competition is at no cost other than time and travel and is open to any 
public or private school or homeschool group that registers. A teacher is required as coach for school teams as the 
point of contact for the organizers. Participating schools enjoy the same flexibility in the implementation of GEAR 
at their school as we have in the organization of the tournament. They decide on the selection of participants, 
meeting times, total number of teams and participants (there is an upper limit of 10 teams per school). They find 
computing facilities at school and organize the trips to campus. Over the years, the organizers have acquired over 
150 LEGO MINDSTORMS NXT kits and each school can check out up to six kits for the duration of the 
competition at no cost. This has allowed schools to evaluate participation in robotics and its benefits before making 
a significant financial investment. It also enables small rural schools and Title 1 schools that would otherwise not 
have the resources to participate to do so. 

In addition, schools in proximity to the university campus have the option to request engineering mentors to 
assist their teams. Mentors are engineering freshmen enrolled in a service learning section of ENGR 1315: 
Introduction to Engineering, a college wide freshman level introductory course. Descriptions of service learning 
courses for engineering freshmen can be found in Osborne, Thomas, & Forbes (2010) and Karp (2011). 

3 METHODOLOGY AND DATA 

Data for this paper were collected through multiple qualitative and quantitative means: participant 
observation; registration forms; team demographic reports that required open-ended answers to questions; interviews 
with students, engineering undergraduate mentors, and teachers; and a pre/post-test survey that measured student 
attitude shifts over the duration of the program. Schools were all located in the West Texas region, see Figure 3 
below, and opted into this program after hearing about it through notifications via email, through information 
sessions conducted by the organizer, and through the teachers' interpersonal socio-professional network. 

In 2011 and 2012, team information was collected from the coaches. In 2011, 31 of 35 schools submitted 
their team information sheet reporting 511 participants, and in 2012, 38 of 49 schools reported 471 participants. 
Since the inception of the program at TTU in 2006, over 60 schools (28 elementary and 35 middle schools) chose to 
participate, this allowed for an overall student sample of over 1500 participants. Undergraduate mentors were 
initially attracted through payment, then, in subsequent years, were compensated through a freshmen-level service¬ 
learning course credit of a required engineering class (Karp, 2011). Over six years, over 150 engineering 
undergraduate mentors participated. Engineering undergraduate students who served as team mentors were 
expected to complete 15 hours with their school teams and document their mentoring experience in a journal. They 
also had to write three reflection papers in which they evaluated skills required in engineering (soft and hard), and 
the connection between GEAR tasks their teams perform and the engineering profession. 

Interviews and participant observation were conducted during practices and competitions. Written data, 
such as the team reports and registration forms were collected at the beginning and end of the program. In 2010 


' National Robotics Week, http://www.nationalroboticsweek.org/ , accessed February 2013. 

10 GEAR General Documents, http://www. gearrobotics.org/im/index.php?option=com wrapper&view=wrapper&Itemid=71 , 
accessed February 2013. 
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(year five of conducting this program), we wished to have a formal metric of student learning and attitude shift over 
the course of the program and so decided to conduct a pre- and post-test survey of participants. This survey was 
designed by Nugent et al. (2009) and students were asked the same 25 questions at the beginning and the end of the 
program. All answers were entered on a Likert scale ranging from ‘strongly agree’ (5) to ‘strongly disagree’ (1). 
Participation was voluntary, and 16 of the 24 schools that were in the program that year completed both the pre- and 
post-test. Of the approximately 400 students who were enrolled in the program in 2010, 300 took the pre-test and 
225 of that 300 took the post-test, thereby resulting in an overall response rate of over 50%. As part of the survey 
we collected the students’ gender, ethnicity, socio-economic status, grade level, and prior experience with LEGO 
robotics (Karp & Schneider, 2011). 

Data were analyzed through multiple means. The qualitative data were analyzed through grounded theory 
(Glaser and Strauss, 1967; Charmaz, 2006) whereby themes emerged holistically from repeated reviews of the 
verbal and written responses of teachers and students. These themes were then tested through specific questions 
asked to students, teachers, and undergraduate mentors for accuracy and authenticity. The quantitative data 
(gathered from formal surveys, team demographic sheets, and registration forms) were entered into data analysis 
software, and then analyzed through paired two-tailed paired t-tests to determine the program's effect on students 
and student sub-groups (for specifics and results, see section C in Chapter 4 below). 

4 EVIDENCE OF SUCCESSFUL IMPLEMENTATION OF AN AFTERSCHOOL ROBOTICS 

PROGRAM 

The best planned program is useless without data that supports its efficacy. That is, to determine the 
usefulness of this program, we need to know to what extent GEAR participation is attractive to teachers, engineering 
undergraduate mentors, and participants and what effect it has on the latter. Generally, through growth in 
participation over the six years of the program's existence, team demographics, and teacher and engineering 
undergraduate mentor perceptions, we find that the program excites traditionally underrepresented student 
participants about STEM and engineering in particular and teaches both K-8 and undergraduate students valuable 
skills such as teamwork, problem solving, and critical thinking. In the short term, our data indicate that our 
afterschool robotics program is indeed a positive asset for multiple educational stakeholders: students, teachers, and 
undergraduate engineering majors. 

A. Growth in Participation 

One criterion to measure the attractiveness of a program is by evaluating the number of participants in the 
program over time. Our local GEAR tournament has seen continuous growth over the last seven years, both in 
terms of number of schools participating as well as in number of individual participants. It was implemented in 
2006 as part of a Texas Technology Workforce Development Grant 11 which aimed at improving retention of 
electrical engineering students by offering them paid mentor positions at an early point in their degree program 
(freshman and sophomore year). The challenge was first piloted at a single elementary school with 20 students on 
four consecutive Saturdays to evaluate its feasibility and to gain some experience. The first non-pilot GEAR 
tournament was offered to local elementary schools in 2007 with 12 participating schools all of which participated 
again in 2008. In 2009, participation was opened to middle schools based on requests from middle school teachers 
who had heard about it from their elementary school colleagues, parents of 6 th graders who participated in GEAR in 
elementary school, or through the local media. Since 2010, separate competitions for elementary schools and 
middle schools have been held in order to accommodate the increased number of participants. While we sent out 
flyers to local schools in 2007 and expended the number of schools participating significantly by advertising the 
event through electronic newsletters sent out by the Texas Education Service Centers in 2011, many first-time 
participating teachers heard about the competition from their colleagues at other schools. 


11 Texas Higher Education Coordinating Board Technology Workforce Development Program, (2006), 
http://www.thecb.state.tx.us/index.cfm'?obiectid=9B6BC3F5-F808-5745-885FDE77874A96E6 . accessed February 2013. 
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Table 1 shows the number of schools and number of teams that participated in our GEAR competition each 
year separated by school type: elementary schools (K-5 th grade) and middle school (6 th - 8 th grade). As mentioned 
above, GEAR was not offered to middle schools until 2009 and in that year a joint competition was held for both 
school types. From 2009 on a constant growth in the number of participating schools can be observed from the 
table. It is graphically shown in Figure 1. The decrease in participation from 2008 to 2009 is due to the transition 
made in LEGO robotics kits at that time. Some schools which already owned the older LEGO RCX kits never 
started using the at-the-time newly introduced LEGO MINDSTORMS NXT kits and stopped participating. Since 
each school could participate with up to 10 teams. Table 1 also shows the growth in number of participating teams 
for each year. While there are no restrictions on team sizes, observations at the competitions have revealed that 
most teams consist of 2-4 members. Some school districts had teams with members from grades 4-7, thus the 
uncertainty in number of elementary and middle school teams for 2009. While we originally started by offering 
GEAR to elementary schools only, its recent growth both in number of schools as well as number of teams is among 
middle schools. Information from teachers provided on the registration forms reveals that many middle schools 
have ways to integrate GEAR into robotics or technology classes and that those teachers had experience with LEGO 
robotics prior to participating in GEAR. As motivation to participate in GEAR these teachers often mentioned the 
application of robotics tasks that the competition theme provides and the added competitive component. 


Table 1: Number of schools and teams participating in the TTU GEAR competition by year 


Year 

Elementary Schools (K-5) 

Middle Schools (6-8) 

Total # of teams 

# of schools 

# of teams 

# of schools 

# of teams 

2007 

12 

not available 

not offered 

N/A 

not available 

2008 

13 

not available 

not offered 

N/A 

not available 

2009 

7 

30-39 (Some mixed 
teams) 

6 

11-20 (Some mixed 
teams) 

50 

2010 

9 

50 

13 

47 

97 

2011 

16 

75 

19 

76 

151 

2012 

20 

88 

29 

129 

217 



Figure 1: Number of schools participating in our GEAR competition 


Since its inception, 28 different elementary schools and 35 different middle schools have participated in the 
annual TTU GEAR competition. Of these schools, 66% of the elementary schools and 54% of the middle schools 
are classified as Title 1 schools, see Table 2, which further splits Title One schools into three different categories 
based on the percentage of low income students at these schools (<70 %, 70% - 90%, >90%). Our competition can 
thus be considered as successful in attracting students from underserved populations. The majority of Title 1 
schools checked out LEGO MINDSTORMS NXT kits from us and feedback from teachers confirmed that they 
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would not have been able to participate if there was a registration fee associated with the competition or if they had 
to invest in their own LEGO kits. This indicates that, to gain or maintain a presence in Title 1 schools, monetary 
barriers to entry such as registration fees or payment for kits should be low or non-existent. 

Table 2: Percentage of Title 1 schools that participated in GEAR robotics and percentage of low income student 


population in Title 1 schools (data provided by the Texas Higher Education Agency 12 ) 



Elementary Schools (K-5) 

Middle Schools (6-8) 

Non-Title 1 schools 

34% 

46 % 

Title 1 schools (total %) 

66% 

54% 

<70 % low income student population 

33 % 

37% 

70 % - 90 % low income student population 

22 % 

17 % 

>90 % low income student population 

11 % 

0% 


Apart from being able to attract schools to GEAR, annual repeated participation is another indicator of 
offering a program that teachers and participants consider valuable. Most schools participate in the GEAR 
competition on an annual basis. Over the years, eight elementary schools stopped participating GEAR. In five cases 
the school administration had decided that they would not continue in the competition any longer (reasons are 
unknown), in two cases the robotics teacher retired and nobody took over that responsibility, and the reason for the 
last case is unknown. Out of the six middle schools that stopped participating, two cases were due to lack of interest 
among students (too many other conflicting activities students were engaged in), two due to teacher retirement, in 
one case a combined elementary/middle school moved participation from middle school to elementary school, and 
the last case was a one-time participation of a group of students who had done GEAR in elementary school and 
participated again their first year in middle school. In summary, a large number of schools annually participate in 
GEAR (71 % of all elementary schools and 83 % of all middle schools). 

Figure 2 shows for how many consecutive years the schools that participated in the 2012 competition have 
been participating in GEAR (divided into elementary schools in the top graph and middle schools in the bottom). It 
also provides information about how far the schools are away from the tournament location, split into three 
categories: less than 15 miles, 15 to 50 miles, and over 50 miles. The one elementary school listed as “other” in 
Figure 2 participated in 2010 and 2012 but not in 2011 due to health issues of the robotics coach. 
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12 Texas Education Agency (TEA), Title 1 Campuses 2009-2010, 

http://www.tea.state.tx.us/index2.aspx?id=%202147499694&menu id=798&menu id2=2147483722 , accessed February 2013. 
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Figure 2: Number of consecutive years 2012 GEAR schools participated in the program, separated by school’s distance 
from tournament location; top: elementary schools, bottom: middle schools 


As can be seen from Figure 2, many of the schools that participate have been in the program for the whole 
of the program's existence: 6 schools of the possible 20 elementary schools have participated for six consecutive 
years and 5 schools of the possible 29 middle schools have participated for 4 consecutive years. The majority of 
schools are either first- or second-time participants or long-time veterans (as can be seen by the dip in the middle 
years of participation). We attribute this dip to schools deciding not to participate due to the new LEGO kits that 
came out about four years ago and to a lack of advertisement in those years. This is further evidence that these 
LEGO kits may need to be subsidized, particularly for Title 1 schools. 


It is interesting to note that the majority of first-time participating schools (as of 2012) are from over 50 
miles away (see Figure 3 for the geographical region participating schools are from). For middle schools, this also 
holds true for schools that participated since 2011. Most of these schools already had robotics programs in place 
and were looking for applications and/or challenges to incorporate into these LEGO robotics programs and to 
motivate their students. This may suggest that this program has become more widely known in the surrounding 
region, as well as indicate a future need for a locally-based program that may not require the students to travel as far. 
Since those K-8 students who are more than 15 miles away do not benefit from the coaching of the engineering 
undergraduate mentors (due to time and travel constraints for the undergraduates), in the future, we may look into 
video conferencing alternatives to connect engineering undergraduate mentors with participants. However, this 
option may be limited by the lack of appropriate webcams and internet access in low-income elementary and middle 
schools as well as other factors like the inability of the undergraduate mentors to interact with participants on a 
personal level or physically manipulate objects in front of them. Due to the fact that West Texas is relatively rural 
and geographically spread out, this may be a limiting factor inherent in this area alone and may not be faced by other 
programs in other geographic regions. 


Comparing Figure 1 with Figure 2, one notices that six of the twelve initially participating elementary 
schools (50%) have participated every year from 2007-2012. The same holds true for five of the six middle schools 
that participated in GEAR in 2009. In all these cases the participants were coached by the same teacher all those 
years, thus suggesting that the program's relationship with the school over time and professional relationships with 
specific teachers matter for continued participation. 


While we can demonstrate growth in participation by number of schools and numbers of teams, as shown 
in Table 1 and Figure 2, the only longitudinal data that were collected on an individual participant basis was the pre- 
and post-test survey during 2010 (results are discussed below in section C). Given the flexibility of implementation 
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at the school level (a criterion that we consider as one of the success factors of our program), many different 
constraints exist with respect to who gets to participate in the program. Some schools limit participation to a single 
grade level; others have developed a sophisticated application process including tryouts. A general consensus 
among GEAR coaches is that students participating in GEAR must have good academic standing, and interest in 
robotics, and be available for after school meetings. Thus, given the variance in selection processes and student 
populations between schools, it is likely that this program's effects are mediated by school setting and student 
demographics. 

Figure 3 shows the geographic region in West Texas of schools that participated in GEAR in 2012. While 
11 of the 20 elementary schools (55%) were located within a 15 miles radius of the Texas Tech University campus, 
only 8 of the 29 middle schools (24%) were located within this radius and 19 (66%) traveled over 50 miles, two of 
them (7%) close to 300 miles, in order to participate in the competition, thus signaling their commitment to the 
program and their assessment of the program's importance to their students. 


7 

% % ~ 



Figure 3: Location of schools that participated in our GEAR competition 2012 
B. Team Demographics 

Although no longitudinal data over the course of the entire six year program were collected, data from the 
pre- and post-test survey as well as the team demographic forms allow us to see a snapshot of student and school 
demographics from 2010-2012. Tables 3 and 4 summarize the information obtained from these sources. 


Table 3: Gender and ethnicity of participants (PI: Pacific Islander) 


Year 

Male 

Female 

African Amer. 

Amer. Indian 

Asian/PI 

Hispanic 

White 

Other 

2010 

56% 

44% 

7.1% 

< 1 % 

1 . 8 % 

36.4% 

47.6% 

6 . 2 % 

2011 

67% 

33% 

4.7% 

< 1 % 

2 . 2 % 

37.4% 

48% 

7.6% 

2012 

67% 

33% 

3.8% 

< 1 % 

2 . 1 % 

32.1% 

55.2% 

6 . 6 % 


Table 4: Grade level distribution of participants 


Year 

2 nd 

3 rd 

4 th 

5 th 

6 th 

fyth 

8 th 

2010 


1 . 8 % 

14.7% 

24% 

26.7% 

17.8% 

14.2% 

2011 

0 . 6 % 

4.9% 

12.5% 

30.9% 

19.2% 

16.6% 

13.7% 

2012 

0.9% 

1.5% 

18.6% 

23.7% 

17.5% 

25.2% 

11.7% 


In 2010, students self-identified into racial groups. In 2011 and 2012, teachers categorized students by 
race. We believe that the drop in Hispanic student participation in 2012 is due to the missing team information 
sheets from 12 schools, many of which reported a high Hispanic population participating in GEAR in previous 
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years. Each year, we observe a large number of female participants often organized in “all girls” teams. Table 3 
reflects that about 1/3 of the participants are girls in all three years. In 2010, the percentage of girls was even higher 
(44%). In that year, however, one middle school, which had implemented GEAR as part of a required class, 
contributed data for all their students involved in robotics and not just the participants of the competition, which 
raised the number of participating females. Due to a lack of data, we cannot determine which students participated 
in competitions and in the program during school hours as opposed to those who participated in the program during 
school hours but did not participate in the competitions. 

Table 4 shows that the majority of participants are from grade levels 4 through 7. Students in grade levels 
K-2 typically do not have the necessary fine motor skills for the LEGO MINDSTORMS NXT kits. Third graders 
are often not considered due to the limited number of slots for participating. We have also observed that 8 th graders 
often move on to more elaborate robotics competition such as BEST (Boosting Engineering, Science, and 
Technology) 13 , if offered at their school. 

C. Pre- and Post-Test Survey 

As mentioned in the methods section, we conducted a pre-test and post-test survey designed by Nugent, 
Barker, Grandgenett & Adamchuk (2009) during the 2010 GEAR challenge to evaluate a LEGO robotics program 
for the same age group in Nebraska. The findings from this survey indicate that our robotics program is especially 
helpful for female participants, non-white Hispanics, and those who report that they receive a free school lunch, thus 
indicating low familial socioeconomic status. The survey consisted of 25 questions that could be grouped into three 
different concepts (Karp & Schneider, 2011): 

I. Attitudes toward the scientific method of investigation 

II. Attitudes toward robotics and LEGO robotics in particular 

III. Attitudes toward working in groups 

A two-tailed paired sample t-test with significant level of 0.05 performed on all 225 participants and then 
on specific subgroups resulted in significant self-reported changes in attitude over the course of the program for 
some of the following questions: 

• Question 2 (concept II): It is important for me to learn about robotics 

• Question 7 (concept I): It is important for me to collect and interpret data to verify a prediction or 

hypothesis. 

• Question 11 (concept I): I like using scientific methods to solve problems. 

• Question 17 (concept I): I carefully analyzing a problem before beginning to develop a solution. 

• Question 20 (concept II): I am certain I can fix the software program for a robot that does not behave as 

expected. 

• Question 21 (concept II): I am confident that I can program a LEGO robot to follow a black line using a 
light sensor. 

• Question 23 (concept III): I like to be part of a team that is trying to solve a problem. 

Using data from all participants, a significant increase was seen for questions 7, 11, 20, and 21, indicating 
an increased confidence in successfully solving robotics tasks at the end of the program and a higher appreciation 
for the scientific method of investigation. For specific coefficients and standard deviations (as well as precise 
significance levels), see Karp & Schneider (2011). 

Next, we looked at typically underrepresented groups. When performing the paired sample t-test for boys 
and girls separately, we saw a significant change toward a higher score for questions 11, 20, and 21 for girls while 
no significant change was seen among boys for any of these questions. This indicates an increase in affinity for 
scientific methods and increase in robotic confidence in females, a group typically underrepresented in this field. 


13 Boosting Engineering Science and Technology (BEST), http://www.bestinc.org/ . accessed February 2013. 
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When comparing Hispanic students against white students (not enough data was available for any of the other ethnic 
groups) a significant increase was observed among Hispanic participants for questions 17, 20, and 21 but not for 
white participants. Again, this indicates an increase in confidence and in problem analysis as part of scientific 
investigation. The only significant change in attitude that was found among white participants was a decrease in 
reporting enjoying solving a problem as a team - an unanticipated result. When looking at the same question result 
for girls, they actually scored higher in the post-test than in the pre-test but the change was not significant (p <.05). 

We also compared students who qualified for a free lunch program at school (low socio-economic 
standing) with those who do not. A significantly higher score was seen among the low socioeconomic status group 
for questions 7, 11, 17, and 21 in the post-test than in the pre-test. For the non-free lunch students, the only 
significantly higher score on the post-test than pre-test was on question 20. This suggests that this program may 
cause an increase in understanding the importance of the scientific method as well as an increase in confidence for 
students from low-socioeconomic backgrounds. We were puzzled by the lack of significant increases in students 
from high socioeconomic backgrounds in confidence and importance of the scientific method, so asked teachers 
what they thought this might be due to. Many teachers suggested that the high socioeconomic students already had 
LEGO sets and other similar toys at home, so would enter the program already feeling confident in their ability to 
manipulate such toys. Last, in an effort to evaluate the longer term attitudinal changes among participants, we 
compared first time participants with participants with prior robotics experience. Among all subgroups, only 
participants with prior robotics experience showed a significantly increased perception of robotics being important 
to them, suggesting that the opportunity for repetitive exposure to robotics is important to achieve a long-lasting 
interest in STEM. 

In summary, only a few questions resulted in a significant change in self-reported attitude between the pre¬ 
test and post-test survey. However, most positive changes were observed among participants who are typically 
underrepresented in STEM fields. 

5 BENEFITS OF GEAR PARTICIPATION ACCORDING TO TEACHERS 

Our intention in offering GEAR is to expose young students at an early age to engineering through hands- 
on challenging activities that promote critical thinking, the engineering design process, application of sciences, and 
teamwork at an early age. Through the annually changing GEAR challenge and invited speakers at the on-campus 
kickoff event, we promoted engineering careers and a college-going mindset. This is particularly important for 
participants who do not have engineers or even college graduates in their family. According to teachers and 
participants, the GEAR events are often their first exposure to a university campus in an academic setting. 

In our 2011 and 2012 team demographics form, we asked teachers open-ended questions to determine what 
they consider as benefits in participating in GEAR. Additionally, we spoke directly with teachers regarding what 
they considered the benefits and impacts of this program were on participating students. For both years and 
elementary and middle schools combined, we received open answers from 46 (21 elementary and 25 middle) 
teachers over the course of two years. According to the principles of grounded theory (Glaser & Strauss 1967; 
Charmaz 2006), the answers were analyzed for common themes, then the frequency of each theme was determined. 
These concepts were not necessarily mutually exclusive - for example, a teacher might speak about the importance 
of learning how to persevere in problem-solving. This would be counted as both "problem-solving" and 
"perseverance." The results of the written answers are shown in Table 5. The teachers who participated in this 
program were universally enthusiastic about its impact on students and critical thinking skills: 

The main benefits of participating in GEAR are as follows: practice and implementation of problem solving, 
communication, and team building skills. It also gives the students an opportunity to see places and have 
experiences they would normally not be able to, i.e. visiting [Texas Tech University] and working with robots. 

By far, the most commonly teacher-reported benefit of participating in GEAR was an improvement in 
student teamwork skills, followed by problem solving and critical thinking, and then computer programming skills. 
Interestingly, other than computer programming, the application of math, sciences, engineering, and technology 
concepts was rarely mentioned by teachers as a benefit of GEAR participation, although many teachers noted that 
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GEAR "recruited" students into these areas. Soft skills, such as persistency, time management, the challenging 
aspect of a competition, creativity, and the creation of a college going mindset were more commonly perceived as 
benefits, especially for Title 1 students. As one teacher put it: 

Teamwork exposes students to higher order thinking skills, technology, science, and math. [It also is] exposing Title 
1 students to a world of experiences that they would not have at any other place. It also is recruiting students from 
the 4th [grade] in the science and math areas. 


Table 5: Benefits of participation in GEAR according to teachers 


Benefit 

Occurrence 

teamwork / cooperative learning 

40 

problem solving 

34 

critical thinking / higher order thinking 

13 

introduction to computer programming 

13 

challenge/competition 

9 

new experiences / hands on activity / skill building 

9 

Creativity 

8 

travel / campus experience 

8 

persistency, perseverance, patience, tenacity, dependability 

7 

design / construction 

6 

introduction to technology/technology skills 

5 

gets them excited about STEM 

4 

exposure to engineering 

4 

self-esteem / confidence / success 

4 

prepare for future 

4 

application of science concepts 

3 

using math skills 

2 

outside the box thinking 

2 

competition at no cost 

2 

engineering process 

2 

Excitement 

2 

Motivation 

2 

improvement in attitude and discipline 

2 

planning / meeting deadline 

2 

learn to reflect / observation of others 

2 

communication skills 

2 

Fun 

1 

trial and error 

1 

working with robots 

1 


Research performed by Duncan-Wiles, Jones, Diefes-Dux, & Brophy (2012) regarding the implementation 
of engineering in the elementary and middle school curriculum has revealed a reluctance of elementary and middle 
school teachers to talk about engineering in class even after participation in teacher professional development. The 
authors state that while the teachers are implementing the engineering related activities covered in a professional 
development workshop, they rarely relate these activities to the field of engineering or the engineering design 
process (Duncan-Wiles et al. 2012). The fact that the teachers in our study predominantly spoke about critical 
thinking skills and team work and only mentioned “exposure to engineering” four times may be reflective of this 
reluctance. To calm those fears and provide an extra adult in the classroom with some expertise in engineering, the 
engineering undergraduate mentors bridge this gap. 

6 THE ROLE OF ENGINEERING MENTORS 

Providing engineering undergraduate students as mentors to local GEAR teams has turned out to be another 
success factor in attracting schools to participate in GEAR as well as in the program itself. Various successful 
formats of selecting engineering mentors have been tested during the last years. We originally started by offering 
paid mentor positions to electrical and computer engineering students as part of our Texas Workforce Development 
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grant. 14 Since 2010, a Service Learning section of ENGR 1315 Introduction to Engineering is offered every spring 
semester and mentoring local school teams is part of the service learning assignment (Karp, 2011). ENGR 1315 is a 
freshman level engineering course required by most engineering majors and pre-engineering students and is 
typically taken during their first year on TTU campus. Students in this service-learning section must choose to be a 
part of it and receive additional training in the university classroom concerning how to interact with K-8 students as 
well as how to work with the LEGO kits. 

For first-time participating teachers, the availability of a mentor takes away a large part of the intimidation 
caused by the perceived requirement of being a robotics expert and technologically savvy. Engineering 
undergraduate mentors and teachers quickly form a symbiotic relationship: while the teacher organizes the 
classroom setting, selects participants, deals with behavioral and discipline issues, communicates with parents, and 
deals with other housekeeping tasks, the engineering undergraduate mentor becomes the technology and engineering 
expert who is responsible for software installation, and answering any type of questions regarding programming or 
robot design. Teachers from local schools annually request mentors for their teams. While some teachers only want 
one engineering undergraduate mentor for all their teams, others assign an engineering mentor to each of their 
teams. Teacher feedback on the team demographics forms does reveal that the performance of the mentors and 
quality of the mentoring varies by undergraduate student, but the majority of teachers were quite positive and 
grateful for their mentors. Some noted that having mentors of similar race to the student population was helpful in 
providing role models to students. Others were thankful for the extra hands and expertise in the classroom: 
"mentors were great and most needed because I have a large group." 

Teachers annually request mentors to come to their school even if last year’s mentors were only moderately 
helpful. No teacher has ever requested that no engineering mentor should come to their school. Even teachers who 
are robotics veterans and have participated in the program for several years appreciate the presence of the 
engineering mentors in the classroom. Having more adults in the classroom allows them to enroll more students in 
the program. “Mentors go a long way,” said one veteran robotics teacher. He continued: 

Students are more invested when their team has a mentor. Mentors are not too much different from the students and 
show them that they also can go to college one day. When having a mentor for each team, the competition is not 
only among the teams but also between the mentors, taking the competition to a whole different level and resulting 
in a larger variety of robot designs. Teams that have mentors practice explaining what they’re doing to adults and 
perform better in the competition, when being interviewed by or negotiating with judges. 

Another teacher who was particularly pleased with the 2012 mentoring experience wrote: 

These guys were great! It was a bonus that they were also minorities. 1 always think it’s great when our children 
get to see people “like them ” being successful. They helped get the kids to think and did not just give them the 
solutions. This was an extremely positive experience for all involved this year. They were positive with the kids and 
talked to them appropriately. They were on time and if they couldn’t make it they always let me know ahead of time. 

Student mentors were equally positive about the experience, reporting that they enjoyed spending time with 
children and applying the skills that they were learning on the university level. Some reported that teaching the 
basic skills actually solidified higher-level concepts in their own minds, thus contributing to their own learning. 
Since students were receiving course credit for participating, it seems possible that they were also so positive about 
the experience when asked because their professor(s) were the ones questioning them. However, many have 
approached us after the course was completed to repeat that they "really enjoyed working with the school teams and 
learned something." Some students - particularly females - stay involved in the program in following years either as 
mentors or as hourly paid undergraduate assistants. Overall, we argue that mentoring elementary and middle school 
students during the eight week GEAR LEGO robotics competition teaches undergraduate students communication 
skills, critical thinking, problem solving, team work, professionalism, and leadership, i.e. skills that are hard to teach 
in a university classroom setting. 


14 Texas Higher Education Coordinating Board Technology Workforce Development Program, (2006), 
http://www.thecb.state.tx.us/index.cfm?obiectid=9B6BC3F5-F808-5745-885FDE77874A96E6 . accessed February 2013. 
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7 POSSIBLE LIMITATIONS 

While we overall found that the effects of this program are positive for all involved (the K-8 students, 
teachers, and undergraduate mentors), there are some limitations in this study that could be addressed in future 
research. First, students and teachers self-select into this program, possibly because of prior interest in robotics 
and/or engineering. Thus, excitement and knowledge about engineering and scientific skills may not be only due to 
participation in this program. And, while we considered the flexibility and adaptability of this program to be a 
strength in recruiting schools and participants, the fact that it was used differently in different schools (during school 
hours as opposed to an after-school program) and different schools had different criteria for students to participate 
(everyone who applies versus a "tryout" system) may limit the comparability of data between schools. Regarding 
effects on teacher and student knowledge, we are unsure whether teacher comfort with engineering and scientific 
concepts and student knowledge have been improved, since the majority of teachers focused on the improvement in 
student critical thinking and teamwork skills rather than STEM concepts when asked about the benefits of this 
program. Additionally, teacher and participant response rates to questionnaires/team forms varied between 56 to 88 
%, indicating that our data may not be a complete picture of all who participated in this program. 

8 CONCLUSIONS 

We have described the six year long successful implementation of a robotics program for elementary and 
middle school students that not only benefits the teacher and K-8 student participants, but also college level 
engineering students who serve as mentors to local teams. Some of the causes of this success are the flexibility of 
the implementation at the organizer and the school level, the fact that participation is at no cost, and the synergistic 
relationship between teachers and engineering students. In terms of the impact of this program, it has seen 
significant growth over the six years of its existence and continues to attract a diverse body of participants and a 
large percentage of Title 1 schools. 

Regarding individual-level effects on K-8 students, this program has been shown to have a positive effect 
on the confidence levels and attitudes towards the scientific method and robotics in female students as well as 
Hispanic students. Since these are traditionally underserved populations in the STEM fields, reaching students 
while they are still in early years of their schooling may aid in increasing their populations at the university and 
professional level. Overall, students report enjoying this program and learning about robotics. Students who receive 
free lunch (thus signaling low familial socioeconomic status) also report increased confidence levels as well as 
knowledge and use of the scientific method. Teachers report that their students have improved on critical thinking 
and teamwork skills, and also state that they enjoy having the program (and its undergraduate mentors) in their 
classrooms for role-modeling purposes for the K-8 participants. Additionally, this program has had a positive effect 
on those undergraduates who are already training to be professional engineers in terms of ability to communicate to 
different audiences, knowledge formation, and professional skills such as punctuality and preparation. They also 
report that their knowledge of the some fundamental concepts is improved by having to teach them in detail to 
younger students. 

Overall, sustainability of this program beyond the initial grant funding period was achieved by exploiting 
its educational opportunity and involving engineering freshmen in form of a service learning project. In addition, 
funding from various companies and foundations was obtained, thus suggesting that it is a method for successfully 
melding university service-learning, corporate sponsorship, and knowledge acquisition at the elementary and middle 
school levels. 
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